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I.   INTRODUCTION 

A.   BACKGROUND 

Since  the  mid  1960 's  all  phases  of  the  weapon  systems 
acquisition  process  within  the  Department  of  Defense  (DoD) 
have  undergone  criticism  and  change.   Excessive  life  cycle 
cost  growth  and  poor  operational  performance  of  weapon  systems 
upon  entering  use  have  been  frequent  problems.   As  a  result, 
several  committees  were  formed  to  extensively  study  these 
problems;  among  them  the  President's  Blue  Ribbon  Defense  Panel, 
a  General  Accounting  Office  committee,  and  the  Congress's 
Commission  on   Government  Procurement.   All  have  consistently 
identified  Test  and  Evaluation  as  a  major  problem  area. 

The  acquisition  policy  for  major  weapon  systems  has  evolved 
from  the  high  risk  "total  package  procurement"  philosophy  of 
development  under  a  fixed-price  contract  to  the  more  conserva- 
tive "fly-bef ore-buy"  concept  of  developing  a  new  system  under 
hybrid  cost-type  contracts.   Under  the  latter  concept  program 
risks  and  progress  are  assessed  at  specific  milestone  events 
prior  to  proceeding  into  production.   Concurrency  between 
development  and  production  is  being  minimized  and  greater 
emphasis  is  placed  on  early  determination  of  operational  suit- 
ability prior  to  production  commitments.   This  change  in 
system  acquisition  policy  has  thus  placed  greater  emphasis 
on  the  Test  and  Evaluation  process. 
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The  overall  Navy  Test  and  Evaluation  (T&E)  process  can 
be  viewed  as  a  system  composed  of  testing  agencies  interacting 
in  such  a  manner  as  to  test  and  evaluate  a  weapon  system.   This 
process  is  composed  basically  of  three  types  of  T&E;  Development 
Test  and  Evaluation  (DT&E) ,  Operational  Test  and  Evaluation 
(OT&E) ,  and  Production  Acceptance  Test  and  Evaluation  (PAT&E) . 
Each  type  consists  of  a  heirarchy  of  several  layers  of  organi- 
zational systems;  from  Secretary  of  the  Navy  (SecNav)  down  to 
the  individual  field  activities  under  the  Naval  Air  Systems 
Command  (NAVAIRSYSCOM) ,  or  to  the  squadrons  under  the  Operations 
Test  and  Evaluation  Force  (OPTEVFOR) .   Each  of  the  field  ac- 
tivities can  in  turn  be  subdivided  into  various  systems;  from 
the  Directorate  level  down  to  the  Section  level.   Each  of  these 
layers  of  systems  within  the  types  of  T&E  performed  can  be 
thought  of  as  parts  or  elements  within  the  total  T&E  process. 

The  T&E  process  within  DoD  is  characteristically  bureaucratic 
Chief  of  Naval  Operations  (CNO)  Instruction  3960.10,  DoD 
Directive  5000.3  and  SecNav  Instruction  5000.1,  among  others, 
have  established  the  WHO,  WHAT,  WHY,  and  WHEN  of  the  Navy  T&E 
process.   However  the  question  of  HOW  to  effectively  manage 
T&E  remains  outstanding  and  must  be  addressed.   If  history  is 
any  indication  of  the  future  the  T&E  process  will  become 
steadily  more  complex  as  the  environment  about  the  weapons 
acquisition  process  becomes  more  dynamic,  as  scarce  resourses 
become  more  scarce,  and  as  weapon  systems  become  more  complex. 
The  question  of  HOW  to  manage  the  T&E  process  of  the  future 
becomes  a  perplexing  and  important  issue.   The  manager  of  the 
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future  must  be  able  to  cope  with  the  ever  changing  dynamic 
nature  of  the  environment  about  him  and  also  with  the  com- 
plexities of  the  many  organizational  systems  which  comprise 
the  total  T&E  system  process.   This  thesis  suggests  a  manage- 
rial approach  which  could  cope  with  these  complexities.   This 
approach,  the  systems  approach  to  management,  can  provide  a 
framework  for  thinking  which  guides  the  manager  of  any  organi- 
zational system  to  consider  the  objectives,  environment,  inputs 
and  outputs,  and  interfaces  that  affect  that  system;  and  more 
importantly,  to  view  the  T&E  system  as  a  whole  instead  of  as 
fragmented  parts. 

B.  OBJECTIVES  OF  THESIS 

The  primary  objectives  of  this  thesis  are  two  fold.   First 
is  to  show  that  the  Test  and  Evaluation  process  is  in  fact  a 
system  composed  of  many  interrelated  organizational  systems. 
The  second  objective  is  to  show  that  the  systems  approach  to 
management  offers  a  framework  for  managing  the  Test  and  Evalua- 
tion process  of  the  future  and  as  such  becomes  a  useful  manage- 
rial tool. 

C.  SCOPE  OF  THESIS 

The  systems  approach  to  management  can  apply  to  any  system 
or  subsystem  within  the  total  Test  and  Evaluation  process. 
This  thesis  is  however  restricted  in  scope  to  the  Navy  Test 
and  Evaluation  process  within  the  Department  of  Defense.   This 
thesis  will  draw  from  managerial  experience  gained  during  the 
Test  and  Evaluation  process  of  weapon  systems,  i.e.  F-14  et  al 
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and  relate  the  future  managerial  complexities  of  T&E  to  the 
systems  approach  of  management.   The  overall  concepts  are 
ceraintly  applicable  however  to  T&E  in  general. 

D .   METHODOLOGY 

This  study  is  comprised  of  five  basic  phases,  the  last  four 

of  which  overlap  to  some  extent,  but  are  nevertheless  distinctly 

different. 

Phase  I.     Identification  of  the  problem  and  establish- 
ment of  basic  objectives. 

Phase  II.     Literature  search  in  the  areas  of  Systems 
Management,  Program  Management,  Test  and 
Evaluation,  Government  manuals,  specifications 
and  directives. 

Phase  III.    Information  gathering  from  various  agencies. 
Appendix  A  presents  organizations     inter- 
viewed for  information  pertaining  to  this 
thesis. 

Phase  IV.     Analysis  of  information. 

Phase  V.      Preparation  of  thesis. 

The  process  of  analyzing  Test  and  Evaluation  begins  by  first 
analyzing  the  systems  approach  to  management.   Chapter  II  pre- 
sents the  concepts  and  characteristics  of  the  systems  approach 
and  formulates  a  working  systems  model  which  can  be  applied 
throughout  this  thesis. 

Chapter  III  analyzes  the  Test  and  Evaluation  process  and 
policies  and  delineates  the  basic  types  of  test  and  evaluation 
performed  on  a  weapon  system.   Chapter  IV  analyzes  the  separate 
types  of  T&E  from  a  systems  view  applying  the  systems  model 
presented  in  Chapter  II.   Chapter  V  analyzes  the  total  T&E 
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process  from  a  systems  view.   Chapter  VI  concludes  the  text  by 
relating  the  systems  approach  to  the  management  functions 
within  the  T&E  process  and  suggests  the  importance  of  strategy 
formulation  and  implementation.   Conclusions  then  summarize 
the  basic  findings  of  this  thesis. 
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II.   THE  SYSTEMS  APPROACH  TO  MANAGEMENT 

The  systems  approach  to  management  provides  a  systematic 
framework  of  thinking  about  the  job  of  management  and  stresses 
the  interrelatedness  and  interdependency  of  the  parts  to  the 
whole.   A  systems  approach  has  long  been  observed  by  scholars 
and  practitioners  alike.   One  of  the  earliest  recorded  state- 
ments concerning  systems  and  systems  management  can  be  traced 
to  500  B.C.  when  Mencius  stated: 

"Whoever  pursues  a  business  in  this  world  must  have  a 
system.   A  business  which  has  attained  success  without 
a  system  does  not  exist.   From  ministers  and  generals 
down  to  the  hundreds  of  craftsmen,  everyone  of  them, 
both  skilled  and  unskilled,  use  this  system.   The  skilled 
may  at  times  accomplish  a  circle  and  a  square  by  their 
own  dexterity.   But  with  a  system,  even  the  unskilled 
may  achieve  the  same  results,  though  dexterity  they  have 
none.   Hence,  every  craftsman  possesses  a  system  as  a 
model.   Now,  if  we  govern  the  empire,  or  a  large  state, 
without  a  system  as  a  model,  are  we  not  even  less  in- 
telligent than  a  common  craftsman?"   [George,  1970] 

However,  only  since  the  mid  1960's  has  the  systems  approach 
evolved  into  the  managerial  area.   Appendix  B  presents  the 
evolution  of  management  concepts  and  places  the  systems  approach 
of  management  in  perspective  with  other  classical  theories. 

A.   THE  CONCEPT  OF  A  SYSTEM 

A  starting  point  in  the  analysis  of  the  systems  approach 

is  first  to  define  what  a  system  is.   Noted  authors  in  the 

field  have  defined  systems  as  the  following: 

"A  system  is  an  array  of  components  designed  to  accomplish 
a  particular  objective  according  to  plan."  [Johnson,  1973] 
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"A  system  is  an  organized,  unitary  whole  composed  of  two 
or  more  interdependent  parts,  components,  or  subsystems 
and  delineated  by  identifiable  boundaries  from  its  en- 
vironmental suprasystem. "  [Kast,  1974] 

"A  system  is  an  organized  or  complex  whole;  an  assemblage 
or  combination  of  things  or  parts  forming  a  complex  or 
unitary  whole."  [Cleland,  1975] 

There  are  about  as  many  definitions  of  systems  as  there  are 

authors  in  the  area,  but  all  have  certain  common  characteristics 

The  following  definition  will  serve  as  a  composite  definition 

of  the  term  "system": 

"A  system  is  a  group  of  elements,  either  physical  or  non- 
physical  in  nature,  that  exhibit  a  set  of  interrelations 
among  themselves  and  interact  together  toward  one  or  more 
goals,  objectives,  or  ends."  [Alexander,  1974] 

Diagnosis  of  the  above  definition  of  "system"  will  be  use- 
ful and  will  enable  correlation  with  the  management  function 
within  the  Test  and  Evaluation  process. 

The  word  "elements"  may  refer  to  physical  objects  such  as 
component  parts  of  an  airplane,  or  it  may  also  refer  to  inter- 
related activities  such  as  the  managerial  functions  of  planning, 
organizing,  directing,  and  controlling.   An  "element"  of  a 
system  may  also  be  a  lower-order  system,  hence  a  subsystem  of 
a  larger  system  such  as  T&E  activities  within  the  T&E  system. 
Relations  that  exist  among  system  elements  may  be  examined  in 
terms  of  type  of  relation  present,  whether  spatial,  time,  cause 
and  effect,  physical,  logical,  and/or  mathematical. 

The  purpose  of  a  system  is  shown  by  its  stated  goals  or 
objectives,  the  reason  why  the  system  was  created.   When  a 
system  is  actually  a  subsystem,  or  a  clearly  defined  integral 
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part  of  a  larger  system,  its  purpose  is  usually  to  support 
the  major  system  activity. 

An  important  concept  in  systems  theory  is  that  of  system 
boundaries.   Boundaries  differentiate  between  intra-  and  inter- 
systemic  relations  and  only  when  these  boundaries  have  been 
clearly  defined  can  focus  be  directed  toward  system  elements 
and  interrelations. - 

The  system  of  concern  in  this  thesis  is  the  Test  and 
Evaluation  system;  however  some  mention  of  the  managerial 
systems  within  the  T&E  system  will  be  made.   A  dynamic  model 
will  be  used  to  analyze  the  system  and  its  various  characteris- 
tics.  The  dynamic  model  is  one  of  the  many  types  of  models 
which  can  be  used  to  represent  a  system;  other  models  are 
summarized  in  Appendix  C. 

B.   THE  DYNAMIC  SYSTEMS  MODEL 

Dynamic  systems  are  systems  in  which  certain  elements  are 
altered  in  form  through  an  ongoing,  self-controlling  process 
to  create  other  unique  distinct  elements  [Alexander,  1974] . 
These  systems  are  isomorphic  in  character  and  exhibit  a  pur- 
poseful activity  in  the  achievement  of  their  goals  and  ob- 
jectives.  In  general,  dynamic  systems  display  the  general 
attributes  associated  with  living  things;  purposeful  activity, 
self-regulation,  self-direction,  and  predictable  behavior. 
Inputs  are  transformed  into  outputs  in  a  given  environment. 
A  schematic  model  of  the  dynamic  system  as  used  for  this  thesis 
is  presented  in  Figure  1.   The  significant  features  of  the 
dynamic  system,  as  they  relate  to  T&E,  are  summarized  below. 
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1.  System  Objectives 

Objectives  of  a  system  are  needed  before  the  system 
can  exist,  for  they  describe  the  purpose  and  reason  for  exist- 
ance  of  the  system,  i.e.  the  direction  the  system  is  to  take. 
These  objectives  usually  exist  in  the  form  of  general  and 
specific  objectives.   A  general  objective  may  be  "to  perform 
test  and  evaluation  of  the  total  aircraft".   Specific  objectives 
are  much  more  detailed  and  usually  in  an  exhaustive  list.   They 
could  include  objectives  such  as  "conduct  test  and  evaluation 
of  aircraft  flying  qualities,  performance,  weapons/store 
separation  .  .  . " . 

2 .  System  Inputs  and  Outputs 

An  underlying  characteristic  of  inputs/outputs  is  that 
they  may  be  either  homogenous  or  heterogenous  in  nature.   Inputs 
for  a  dynamic  system  are  defined  as  one  or  more  elements  in  a 
given  dynamic  system  that  are  consumed  or  transformed  during 
the  operation  of  a  dynamic  process.   Outputs  are  defined  as 
one  or  more  elements  in  a  given  dynamic  system  that  are  created 
by  the  operation  of  a  dynamic  system.   Wide  disparity  in  input 
resourses  may  exist  for  the  T&E  system,  such  as  test  airplanes, 
funds,  and  personnel  which  are  used  to  generate  the  various 
outputs  of  reports  and  experience. 

Dynamic  systems  usually  require  a  stream  of  inputs 
which  may  be  either  continuous  or  discrete  over  a  period  of 
time  in  order  to  function  properly.   Thus  the  input/output 
function  can  be  considered  as  a  flow  of  matter  and/or  energy 
over  a  period  of  time. 
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The  relation  between  the  inputs  and  outputs  is  of 
importance  in  describing  the  dynamic  system.   This  in  essence 
means  knowing  the  type  and  quantity  of  inputs  which  are  neces- 
sary to  produce  the  required  outputs.   This  relation  also 
leads  to  determination  of  the  system  efficiency  as  will  be 
mentioned  in  a  later  section. 

3 .  System  Processor 

The  processor  is  the  site  or  environment  where  the 
interaction  among  system  elements  takes  place.   It  is  the 
place  where  transformation  of  the  inputs  to  outputs  occur. 
The  processor  may  have  physical  dimensions  ranging  from  a 
very  small  confined  area  to  a  large  ill-defined  region.   It 
could  also  be  conceptual  in  nature,  existing  only  in  the  minds 
of  men.   Processors  may  be  machines,  equipments,  biological 
units,  the  human  mind,  T&E  activities,  Directorates,  or  Sections 
For  a  T&E  activity  the  processor  is  composed  of  elements  of 
Directorates,  which  are  systems  themselves.   They  are  composed 
of  elements  of  Branches  which  are  composed  of  elements  of 
Sections  which  are  composed  of  elements  of  personnel.   Each 
of  the  systems  and  subsystems  are  a  part  of  the  total  T&E 
system  process. 

4 .  System  Controller 

The  controller  is  that  element  in  the  dynamic  system 
that  controls  the  flow  of  inputs  to  the  processor  and  estab- 
lishes the  mode  of  operation  for  the  processor.   The  controller 
determines  the  way  the  system  will  meet  its  objectives.   He 
can  do  this  by  providing  guidelines  and/or  rules  by  which  the 
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system  must  operate  to  achieve  predictable  output  of  satisfactory- 
quality  and  quantity.   Thus  the  ultimate  performance  of  the 
dynamic  system  depends  to  a  great  extent  on  the  controller's 
design  and  operation. 

The  controller  has  two  separate  yet  interrelated  types 
of  activity.   He  first  regulates  the  flow  of  inputs  in  accor- 
dance with  predetermined  input/output  relations  to  maintain  a 
stable,  dynamically  balanced  system.   This  could  include  such 
actions  as  staffing  the  T&E  activity  to  appropriate  levels 
with  the  proper  technically  competent  people.   Secondly  the 
controller  establishes  the  processor's  overall  operating 
characteristics  by  structuring  the  procedures  and  activities 
to  be  used  in  the  transformation  process.   This  could  include 
setting  certain  quality  and  quantity  standards  for  work  per- 
formance. 

5 .   System  Control  Loop 

The  control  loop  encompasses  the  entire  process  from 
input  flow  to  output  flow  through  control  by  the  controller. 
Needless  to  say,  the  control  of  a  dynamic  system  depends  on 
the  functional  quality  of  the  entire  control  loop.   All  elements 
of  the  system  must  function  properly. 

The  quality  of  the  control  loop  performance  depends 
on  the  information  derived  from  systemic  outputs.   For  system 
effectiveness  those  attributes  of  the  output  stream  that  are 
most  important  in  determining  the  degree  of  achievement  of 
system  objectives  or  those  that  define  the  system  operating 
characteristics  are  the  ones  that  must  be  measured. 
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The  type  of  control  loop  involved  in  T&E  systems  is 
the  "open-loop  system"  as  opposed  to  the  "closed-loop  system". 
The  terms  "open"  and  "closed"  denote  the  degree  of  interaction 
between  a  given  system  and  its  environment.   Open-loop  systems 
are  those  in  which  human  beings  act  as  the  controller.   This 
type  of  system  lacks  a  substantial  degree   of  self-regulating 
capacity  and  its  overall  stability  is  significantly  diminished, 
Thus  the  managerial  capability  of  the  controller,  i.e.  the 
manager,  plays  an  important  function  in  maintaining  system 
stability  and  effectiveness. 

6 .  System  Feedback 

The  term  "feedback"  can  imply  two  separate  meanings. 
First,  feedback  may  refer  to  the  data  gathering  process,  as 
in  gathering  information  for  the  decision  making  process.   The 
term  may  also  denote  the  direct  modification  of  the  input 
stream  by  the  output  stream  of  the  system  without  direct 
controller  intervention.   The  first  reference  to  feedback 
involves  the  control  loop  of  the  dynamic  system.   Here  the 
controller  initiates  or  has  control  over  the  feedback  process; 
such  as  questions  he  may  ask  or  reports  he  may  request.   Where 
the  controller  has  no  control,  as  in  the  latter  form  of  feed- 
back, the  process  is  more  mechanical  such  as  the  thermostat. 
The  T&E  systems  should  consist  of  the  first  type  of  feedback 
discussed,  that  of  information  gathering  by  the  controller. 

7 .  System  Environment 

The  system  environment  is  composed  of  those  factors 
that  influence  the  operation  of  the  system,  and  can  be 
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characterized  as  either  stable,  dynamic,  or  turbulent  in  nature. 
The  system  cannot  afford  to  ignore  the  environment  if  it  is 
to  survive . 

In  the  Test  and  Evaluation  world  the  environment  is 
rarely  stable,  things  and/or  situations  always  seem  to  change. 
Ocassionally  the  environment  is  turbulent,  as  is  the  case  when 
a  test  airplane  is  lost.   However  for  the  majority  of  the  time 
the  environment  is  dynamic,  ever  changing,  perhaps  even  in  a 
predictable  manner. 

It  is  incumbent  upon  the  managerial  system  to  recognize 
that  external  factors  to  the  system  do  in  fact  affect  the  mode 
of  operation  of  the  system.   Among  the  most  often  encountered 
factors  are  funding  changes,  additional  work  load  requirements, 
and  policy  changes  by  top  management. 
8 .   System  Interface 

System  interfaces  occur  in  a  compound  system  environ- 
ment.  Compound  systems  are  those  which  are  composed  of  two 
or  more  distinct  subsystems  that  operate  together  through  the 
action  of  a  unified  controller  or  manager.   Here  the  processor 
is  subdivided  into  two  or  more  separate  units  that  must  function 
together  to  achieve  the  objectives  of  the  total  dynamic  system. 
This  is  most  evident  in  any  organization  system  such  as  a 
Directorate  which  is  composed  of  Branch  subsystems,  all  under 
the  control  of  a  Director. 

An  underlying  characteristic  of  compound  systems  is 
the  interface  required  with  other  systems.   Direct  interface 
is  the  point  where  the  output  of  one  system  becomes  the  input 
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of  another  system.   The  information  interface  is  another  type 
of  interface  which  may  exist.   In  this  case  the  linkage  between 
the  two  systems  is  informational  rather  than  physical. 

Compound  systems  may  be  either  series  or  parallel  in 
nature.   If  the  output  of  one  system  becomes  the  input  to 
another  system,  the  systems  are  in  series.   Such  is  the  case 
of  NAVAIRSYSCOM  and  the  field  activities.   The  outputs  of 
NAVAIRSYSCOM,  the  task  assignments  and  funding,  become  the 
inputs  of  the  field  activities. 

If  two  or  more  dynamic  systems  are  operating  independent 
of  each  other  and  their  outputs  are  combined  to  furnish  the 
inputs  to  another  system,  the  systems  are  in  parallel..  This 
is  the  case  of  a  matrix  organization  where  the  outputs  of 
various  functional  sections  are  combined  to  furnish  the  inputs 
to  a  Program  Manager. 

9.   System  Efficiency  and  Effectiveness 

System  efficiency  may  be  simply  expressed  as  performing 
effectively  at  minimum  expense  of  resourses.   At  the  Section 
level  this  entails  such  things  as  using  different  resourse 
saving  flight  test  techniques  or  data  reduction  procedures. 
Needless  to  say,  cost  reduction  and/or  cost  minimization  is 
always  of  concern  in  any  system. 

System  effectiveness  is  another  important  parameter. 
As  previously  stated,  every  system  has  a  purpose  or  objective. 
The  measure  of  system  effectiveness  is  directly  related  to  the 
degree  to  which  these  objectives  of  the  system  are  met.   Un- 
fortunately in  the  T&E  world  this  sometimes  is  a  nebulous  area, 
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particularily  where  more  than  one  objective  is  involved.   Such 
is  the  case  when  the  objectives  of  a  T&E  system  may  not  be  in 
alignment  with  the  objectives  of  the  contractors  or  of  OPTEVFOR 
during  the  T&E  process  of  a  weapon  system. 

Two  additional  concepts  must  be  considered  under  systems 
effectiveness.   An  evaluation  criteria  must  be  firmly  estab- 
lished before  any  evaluation  of  the  systems  effectiveness  can 
be  made.   Whose  objectives  are  important  and  in  what  order? 
The  second  concept  to  consider  is  one  of  "to  satisfice".   This 
term  may  be  defined  as  the  satisfaction  or  fulfillment  of  needs 
or  objectives  only  up  to  the  point  where  there  is  insufficient 
motivation  to  expend  further  resourses  to  achieve  greater  satis- 
faction. 

Thus  the  concepts  of  system  efficiency  and  effectiveness 
are  separate  and  distinct.   Together,  however,  these  two  mea- 
sures provide  a  means  for  evaluation  of  the  total  systems  per- 
formance. 

C.   SYSTEMS  CONCEPTS  AND  MANAGEMENT 

The  previous  sections  have  presented  the  concept  of  a 
system  and  the  functional  characteristics  of  the  dynamic 
systems  model.   This  section  will  now  relate  the  systems 
approach  to  the  management  functions. 

1 .   Management  Defined 

Perhaps  everyone  has  his  own  definition  or  conception 
of  what  management  is.   Usually  it  is  stated  in  terms  of 
functions  that  managers  do,  not  what  management  is.   In  the 
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context  of  this  thesis  management  will  be  defined  as  "the  process 
whereby  unrelated  resourses  are  integrated  into  a  total  system 
for  objective  accomplishment"  [Johnson,  1964]  .   Performing  the 
function  of  management  is  of  course  done  by  managers. 

Managers  attempt  to  weld  human,  technological,  and 
capital  resourses  into  an  effective,  cohesive  whole  to  achieve 
the  objectives  of  the  organization.   A  manager  gets  things 
done  by  working  with  people  and  physical  resourses  in  order  to 
accomplish  the  objectives  of  the  system.   He  coordinates  and 
integrates  the  activities  and  work  of  others  rather  than  per- 
forming operations  himself. 

2 .   Management  and  the  Systems  Approach 

The  effective  and  efficient  function  of  a  system  is 
under  the  auspices  of  the  controller  or  the  manager  of  the 
system.   To  effectively  perform  the  function  of  management 
within  the  system  the  manager  must  be  aware  of  what  his  system 
is  and  the  role  he  and  it  play  in  the  overall  "big  picture" . 

A  good  manager  relies  on  ideas,  resourses,  things,  tools, 
and  people  to  achieve  objectives.   He  utilizes  such  managerial 
functions  as  planning,  organizing,  staffing,  directing,  and 
controlling  as  prescribed  by  such  authors  as  Koontz  and  O'Donnell 
[1968]  .   In  the  systems  approach  to  management  the  managerial 
functions  are  not  performed  as  separate  entities,  rather  they 
are  performed  in  conjunction  with  operations  of  the  system  as 
a  cohesive  whole.   Any  and  all  operations  and  functions  per- 
formed by  managers  must  be  oriented  toward  the  objectives  of 
the  organization. 
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In  effect  the  systems  approach  to  management  provides 
the  manager  with  a  framework  for  thinking  about  the  job  of 
management.   This  approach  forces  recognition  of  system  ob- 
jectives, inputs  and  outputs,  the  environment,  interfaces  with 
other  systems,  and  overall  effectiveness.   More  importantly 
however  it  guides  the  manager  to  view  not  only  his  immediate 
system  but  the  "whole"  system.   Management  and  the  systems 
approach  as  related  to  T&E  is  addressed  in  more  detail  in 
Chapter  VI. 

D.   SYSTEMS  SUMMARY 

A  system,  no  matter  how  large  or  complex,  must  have  an  ob- 
jective, a  purpose  for  being.   It  will  be  composed  of  inter- 
related elements  or  subsystems  and  will  consist  of  an  input- 
transformation-output  process.   The  system,  in  the  context  of 
this  thesis,  will  function  as  an  open  system  where  exchange  of 
information,  energy,  or  material  will  occur  with  the  environ- 
ment.  The  system  will  have  boundaries  of  operation,  even  though 
at  times  not  explicitely  defined.   The  system  will  have  inter- 
faces, both  inter-  and  intra-system  and  both  functional  and 
informational  in  nature.   The  system  will  also  have  an  environ- 
ment in  which  to  function.   Thus  the  above  descriptive  narra- 
tive is  not  unlike  any  organizational  system  within  the  T&E 
system  process. 

The  systems  approach  provides  a  systematic  way  of  thinking 
about  the  job  of  management  and  provides  a  framework  for 
visualizing  internal  and  external  environmental  factors  as  an 
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integrated  whole.   It  allows  recognition  of  the  proper  place 
and  function  of  subsystems.   It  is  important  to  recognize  the 
integrated  nature  of  specific  systems,  including  the  fact  that 
each  system  has  both  inputs  and  outputs  and  can  be  viewed  as 
a  self-contained  unit.   It  is  equally  important  to  recognize 
that  these  systems  are  a  part  of  a  larger  system. 
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III.   THE  TEST  AND  EVALUATION  SYSTEM 

A.   GENERAL 

The  Test  and  Evaluation  phase  for  the  development  of  a 
weapon  system  is  composed  of  many  organizational  subsystems 
interacting  in  series  and/or  in  parallel,  processing  inputs 
into  outputs  to  obtain  some  objective.   Usually  this  objective 
is  the  timely  deployment  of  a  successful  weapon  system  into 
the  Fleet  in  a  cost  effective  manner.   Figure  2  is  a  much 
simplified,  but  representative,  model  of  the  T&E  process.   As 
such  the  model  displays  all  the  classic  characteristics  of  the 
dynamic  systems  model  as  described  earlier.   The  processing 
function  within  the  T&E  system  is  not  however  composed  of 
actions  and  interactions  with  definable  points  of  interface. 
There  are  role  conflicts  which  tend  to  stretch  out  the  expen- 
diture of  scarce  resourses,  namely  flight  test  time,  and  these 
role  conflicts  eventually  work  there  way  down  the  hierarchy 
of  organizational  systems  and  subsystems  until  they  reach  the 
point  where  the  scarce  resourses  are  physically  managed  and 
expended . 

In  order  to  effectively  analyze  the  T&E  system  and  the 
systems  approach  to  management  of  T&E  the  concept  of  T&E  must 
first  be  presented.   Material  within  this  chapter  is  either 
paraphrased  or  taken  directly  from  DoD  Directive  5000.3,  CNO 
Instruction  3960.10,  the  Navy  RDT&E  Management  Guide,  and 
the  Board  of  Inspection  and  Survey  Aircraft  Test  Directives. 
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MODEL  OF  THE  TEST  AND  EVALUATION  PROCESS 


31 


B.  TEST  AND  EVALUATION  DEFINED 

Test  and  Evaluation  is  an  all  encompassing  pair  of  words 

which  actually  have  a  rather  simple  yet  all  inclusive  meaning. 

The  following  definition  of  T&E,  taken  from  the  RDT&E  Management 

Guide  will  be  used  throughout  this  thesis: 

Test  and  Evaluation  is  defined  as  the  performance  of  tasks 
and  the  evaluation  of  test  results  which  are  conducted 
specifically  to: 

1.  Support  design  and  development  activity. 

2.  Measure  performance  against  specified  acceptance 
criteria. 

3.  Ensure  satisfactory  operation  with  related  inter- 
facing systems. 

4.  Confirm  operational  effectiveness  and  suitability. 

5.  Validate  the  adequacy  of  documentation  for  support 
and  test  equipment,  personnel  training;  maintenance 
and  operation  of  the  whole  system/ subsystems  and 
other  elements. 

The  term  "evaluation"  denotes  the  review,  analysis,  and 
report  of  qualitative  and  quantitative  data  produced  during 
current  or  previous  inhouse  testing,  data  obtained  from 
tests  conducted  by  other  government  agencies  and  contrac- 
tors, from  operational  and  commercial  experience,  or  com- 
binations thereof. 

C.  BASIC  TEST  AND  EVALUATION  POLICY 

Basic  T&E  policy  calls  for  a  development  strategy  based 
on  periodic  performance  demonstrations.   Programs  are  to  be 
structured  and  resourses  allocated  to  ensure  that  the  demon- 
stration of  actual  achievement  of  program  objectives  is  the 
pacing  function.   Test  and  Evaluation  begins  in  the  earliest 
phase  of  RDT&E  with  experimental  testing  of  scientific  hypo- 
theses and  continues  beyond  completion  of  development  where 
primary  emphasis  is  on  perfecting  doctrine  for  the  most  effec- 
tive deployment  of  advanced  weapons. 
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A  basic  T&E  concept  is  that  of  the  "independent  evaluation" 
Distinction  is  made  between  the  Developing  Agency  (DA) ,  i.e. 
the  seller  and  the  operating  forces,  i.e.  the  buyer.   The  DA 
thus  does  not  have  unilateral  control  of  the  establishment  of 
test  requirements,  the  conduct  of  tests,  or  evaluation  of  the 
results.   The  independent  evaluation  is  performed  by  OPTEVFOR. 
Members  of  this  force  must  therefore  perform  a  key  role  in 
determining  test  requirements  and  have  access  to  test  results 
obtained  by  the  DA.   Evaluation  for  service  use,  operational 
effectiveness,  and  suitability  is  performed  by  OPTEVFOR  and  as 
such  have  a  major  impact  on  the  acquisition  milestone  decisions 
These  decisions  are  made  by  the  Defense  System  Acquisition 
Review  Council  (DSARC) . 

The  T&E  process  has  recently  incorporated  the  Test  and 
Evaluation  Master  Plan  (TEMP)  as  a  controlling  management 
document.   The  TEMP  defines  the  test  and  evaluation  plan  for 
each  major  weapon  system  acquisition  program.   It  contains  the 
integrated  test  and  resourse  requirements  of  the  DT&E,  OT&E, 
and  PAT&E  agencies  and  is  updated  periodically  to  reflect  re- 
quired changes. 

D.   TYPES  OF  TEST  AND  EVALUATION 

Department  of  Defense  Directive  5000.3  and  CNO  Instruction 
3960.10  delineate  three  basic  types  of  T&E:   DT&E,  OT&E,  and 
PAT&E,  each  of  which  is  delegated  to  a  different  organization. 
Figure  3  will  aid  in  illustratin  the  relationship  and  time 
phasing  of  each  system  throughout  the  T&E  process. 
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1.   Developmental  Test  and  Evaluation 

Developmental  Test  and  Evaluation  is  planned  by,  con- 
ducted by  or  for,  monitored  by,  and  reported  by  the  DA.   The 
DA  shall  ensure  that  the  DT&E  program  and  the  OT&E  requirements 
are  integrated  into  the  program  schedule  and  with  proper  budgeting 
Also  the  DA  shall  provide  OPTEVFOR  with  all  significant  DT&E 
test  data.   The  objectives  of  this  type  of  test  and  evaluation 
are  to: 

(1)  Demonstrate  that  the  engineering  design  and 
development  process  is  complete. 

(2)  Demonstrate  that  the  system  will  meet  specifi- 
cations . 

(3)  Demonstrate  that  the  design  risks  have  been 
minimized. 

(4)  Estimate  the  system's  military  utility  when 
introduced. 

Developmental  Test  and  Evaluation  is  conducted  in  four 
major  phases.  The  specific  objectives  of  each  phase  are  deve- 
loped by  the  DA  and  published  in  the  TEMP.   The  phases  are: 

(1)  DT-I  is  that  DT&E  conducted  during  the  conceptual 
phase  to  support  the  program  initiation  decision. 

(2)  DT-II  is  that  DT&E  conducted  during  the  validation 
phase  to  support  the  full  scale  development  deci- 
sion.  This  phase,  normally  conducted  at  the  sub- 
system/component level,  demonstrates  that  design 
risks  have  been  identified  and  minimized. 

(3)  DT-III  is  that  DT&E  conducted  during  the  full 
scale  development  phase  to  support  the  first  major 
production  decision.   This  phase  demonstrates  that 
the  design  meets  all  its  specifications.   This 
phase  may  be  further  subdivided  into  other  phases 
such  as  contractor  and  Navy  technical  evaluations. 
The  purpose  of  the  TECHEVAL,  the  final  sub-phase 
is  to  verify  that  the  design  meets  specified  re- 
quirements and  is  ready  for  OPEVAL. 
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(4)   DT-IV  is  that  DT&E  conducted  after  the  first 
major  production  decision  to  verify  that  the 
improvements  or  corrections  of  design  defi- 
ciencies discovered  during  OPE VAL ,  FOT&E,  or 
fleet  deployment  are  effective. 

2.   Operational  Test  and  Evaluation 

Operational  Test  and  Evaluation  is  accomplished  by 
operational  and  support  personnel  of  the  type  and  qualifica- 
tion of  those  expected  to  use  and  maintain  the  system  when 
deployed,  and  is  conducted  in  as  realistic  an  operational 
environment  as  possible.   OT&E  is  planned  by,  conducted  by  or 
for,  and  reported  directly  to  the  CNO  by  OPTEVFOR.   Additionally 
OPTEVFOR  provides  the  DA  with  all  significant  OT&E  test  data 
and  analysis.   The  objectives  of  this  type  of  test  and  evalua- 
tion are  to: 

(1)  Estimate  the  prospective  system's  military  utility, 
operational  effectiveness,  and  operational  suit- 
ability (including  compatibility,  interoperability, 
reliability,  maintainability,  and  logistic  and 
training  requirements) ,  and  need  for  any  modifica- 
tions . 

(2)  Provide  information  on  organization,  personnel 
requirements,  doctrine  and  tactics. 

(3)  Provide  data  to  support  or  verify  material  in 
operating  instructions,  publications,  and  handbooks. 

OT&E  is  subdivided  into  two  major  categories:   Initial 
OT&E  (IOT&E)  which  is  all  OT&E  prior  to  the  first  major  pro- 
duction decision,  and  Follow-on  OT&E  (FOT&E)  which  is  all  OT&E 
after  the  first  major  production  decision.   OT&E  is  divided  into 
five  major  phases  (3  IOT&E  and  2  FOT&E) .   The  specific  objectives 
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of  each  phase  of  OT&E  are  developed  by  OPTEVFOR  and  published 
in  the  TEMP.   The  phases  are: 

(1)  OT-I  is  any  IOT&E  that  may  be  conducted  during 
the  conceptual  phase  to  support  the  program 
initiation  decision. 

(2)  OT-II  is  that  IOT&E  conducted  during  the  vali- 
dation phase  to  support  the  full-scale  develop- 
ment decision.   The  objectives  of  OT-II  are  to: 

(a)  Provide  an  early  estimate  of  projected 
operational  effectiveness  and  operational 
suitability  of  the  system. 

(b)  Initiate  tactics  development. 

(c)  Estimate  program  progress. 

(d)  Identify  operational  issues  for  OT-III. 

(3)  OT-III  is  that  IOT&E  conducted  during  the  full 
scale  development  phase  to  support  the  first 
major  production  decision.   OPEVAL  is  the  final 
sub-phase  of  OT-III.   Specific  objectives  of 
this  phase  include: 

(a)  Demonstration  of  the  achievement  of  program 
objectives  for  operational  effectiveness 
and  suitability. 

(b)  Continuing  tactics  development. 

(4)  OT-IV  is  that  FOT&E  conducted  after  the  first 
major  production  decision,  but  before  production 
systems  are  available  for  testing.   Specific 
objectives  of  this  phase  include: 

(a)  Testing  of  fixes  to  be  incorporated  in 
production  systems. 

(b)  Completion  of  any  deferred  or  incomplete 
IOT&E. 

(c)  Continuing  tactics  development. 
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(5)   OT-V  is  that  FOT&E  conducted  on  production  systems 
as  soon  as  they  are  available.   Objectives  of 
this  phase  include: 

(a)  Demonstration  of  the  achievement  of  program 
objectives  for  production  system  operational 
effectiveness  and  suitability. 

(b)  OT&E  of  the  system  in  new  environments  or 
in  new  applications  or  against  new  threats. 

3 .   Production  Acceptance  Test  and  Evaluation 

This  is  test  and  evaluation  of  production  items  to 
demonstrate  that  the  items  procurred  fulfill  the  requirements 
and  specifications  of  the  procurring  contract  or  agreement. 
This  function  is  carried  out  by  the  Board  of  Inspection  and 
Survey  (BIS) .   BIS  is  responsible  to  the  SecNav  via  CNO  for 
conducting  tests  and  recommendations  for  "acceptance  for  service 
use"  if  test  results  warrant  such.   The  purpose  of  these 
"Service  Acceptance  Trials"  are  to: 

(1)  Discover  and  report  deficiencies  at  the  earliest 
possible  time  so  that  corrective  action  can  be 
taken  to  ensure  that  aircraft  delivered  to  the 
fleet  will  be  capable  of  carrying  out  the  in- 
tended mission. 

(2)  Recommend  design  changes  which  could  improve  system 
effectiveness  and  identify  design  features  which 
should  be  changed  or  avoided  in  future  designs. 

(3)  Determine  if  contract  specifications  and  authorized 
changes  thereto  have  been  satisfactorily  fulfilled 
and  render  an  opinion  as  the  the  government's  or 
the  contractors  responsibility  for  the  correction 
of  defects. 

(4)  Recommend  to  the  SecNav  the  conditions  of  accep- 
tance or  rejection  of  an  aircraft  for  service 
use. 
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E.   TEST  AND  EVALUATION  SUMMARY 

The  T&E  process  is  thus  presented  as  a  hierarchy  of  systems 
all  acting  within  the  overall  T&E  system  process.   The  overall 
T&E  system  is  composed  basically  of  three  large  systems,  the 
DT&E,  OT&E,  and  PAT&E  systems.   Each  of  these  systems  have  ob- 
jectives and  definitive  time  phases  within  the  T&E  process. 
Thus  the  WHO,  WHAT,  WHY,  and  WHEN  of  the  T&E  process  is  provided. 
A  typical  organization  chart  of  the  three  systems  involved  in 
a  weapon  system  program  is  presented  in  Figure  4,  and  it  displays 
an  important  concept  to  recognize  in  applying  the  systems  ap- 
proach to  management.   That  is  as  the  T&E  system  becomes  larger 
and  more  organizations  are  involved  in  T&E  due  to  their  special- 
ization the  number  of  systems  interface  and  combinations  thereof 
increase  enormously. 
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ORGANIZATION  CHART  OF  THE  SYSTEMS  INVOLVED  IN 
TEST  AND  EVALUATION  OF  A  WEAPON  SYSTEM 
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IV.   A  SYSTEMS  LOOK  AT  THE  TYPES  OF  TEST  AND  EVALUATION 

A.   GENERAL 

Modern  T&E  has  come  a  long  way  from  the  scarf  and  goggle 
days  of  screaming  power  dives  and  high-g  pull  outs.   Technology 
in  data  acquisition  systems  and  analysis  procedures  have  kept 
up  with  the  ever  increasing  weapon  systems  technology  and 
complexity.   The  T&E  process  has  grown  from  one  lieutenant  per- 
forming all  flight  tests  on  a  system  to  several  different  and 
distinct  organizations  performing  flight  tests  on  a  weapon 
system  such  as  the  F-14  or  F-18  airplanes.   Growth  and  sophis- 
tication of  the  T&E  process  has  not  however  been  without  problems 
Committees  and  directives  have  provided  guidelines  and  direction, 
but  this  growth  has  often  created  significant  problems;  problems 
which  ultimately  mean  inefficient  and  ineffective  use  of  T&E 
scarce  resourses.   The  following  sections  attempt  to  reconstruct 
some  of  the  more  recent  and  significant  adverse  factors  affecting 
the  overall  T&E  process. 

Recent  theses  by  H.  L.  Young,  Jr.  [NPG  Thesis,  1973]  and 
W.  C.  Bowes  [NPG  Thesis,  1974]  studied  many  aspects  of  the 
overall  T&E  process  as  relating  to  F-14  managerial  problems  and 
to  the  organization  of  T&E  in  general.   Some  of  their  findings 
and  conclusions  are  included  in  the  following  sections  to  add 
more  substantiation  to  the  managerial  factors  affecting  T&E. 

The  following  sections  are  organized  into  the  types  of 
T&E  performed  on  a  weapon  system;  DT&E,  OT&E,  and  PAT&E.   Each 
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of  these  systems  are  subsequently  divided  into  subsection  of 
objectives,  inputs/outputs,  interfaces,  system  environment, 
and  effectiveness,  significant  elements  which  comprise  the 
systems  approach.   A  later  chapter  will  look  at  the  three 
systems  together  in  the  total  T&E  process  system  and  the 
associated  objectives  and  interfaces. 

B.   DEVELOPMENTAL  TEST  AND  EVALUATION 

The  functions  and  operation  of  DT&E  have  been  in  existence 
since  the  first  tests  were  conducted  on  a  Navy  airplane.   Their 
objectives,  interfaces,  and  environment  have  evolved  through 
time  and  can  be  considered  as  "lodged  in  concrete"  in  many 
instances.   Figure  5  provides  a  schematic  model  which  will  aid 
in  the  following  "systems  analysis"  of  DT&E.   The  four  small 
circles  represent  systems  which  are  elements  of  the  larger 
DT&E  system.   The  DT&E  system  is  existing  in  the  environmental 
conditions  as  depicted  on  the  figure.   The  arrows  of  the  indi- 
vidual systems  are  intended  to  signify  only  relative  direction 
of  objectives  with  respect  to  each  other  and  not  magnitude, 
rate,  or  velocity. 

1.   System  Objectives 

The  objectives  of  a  DT&E  organization  are  usually  well 
defined  and  parallel  those  listed  under  paragraph  III-D-1.  Each 
T&E  activity  and  Directorate  within  the  activity  has  mission 
statements  and/or  objectives  of  the  organization  listed.   Addi- 
tionally, work  units  from  NAVAIRSYSCOM  or  other  higher  organi- 
zations supply  specific  direction  and  objectives  for  specific 
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projects.   Finally,  test  objectives  of  the  weapon  systems 
exist  in  the  form  of  specifications  and  performance  guarantees. 

Note  that  Figure  5  implies  that  the  elemental  sub- 
systems comprising  the  DT&E  system  may  have  objectives  in 
varying  directions  with  respect  to  each  other.   Herein  lies 
the  major  problem  that  the  T&E  process  must  face  and  will  be 
treated  in  the  next  chapter. 
2.   System  Inputs/Outputs 

The  inputs  of  the  DT&E  organization  are  well  defined 
in  terms  of  funds,  personnel,  and  test  resourses  such  as  test 
airplanes  or  airplane  subsystems.   However  the  inputs  may  or 
may  not  be  consistent  with  the  overall  objectives  of  the  program 
in  concern.   For  instance,  funds  may  not  be  adequate  or  test 
airplanes  may  not  be  properly  configured  to  make  meaningful 
tests. 

Outputs  of  the  DT&E  system  are  usually  reports  of  test 
results.   Timeliness  and  content  of  reports  have  been  reported 
as  areas  of  significant  concern.  Young  [Thesis,  1973]  and 
Bowes  [Thesis,  1974]  reported  that  project  managers  felt  that 
T&E  reports  are  often  submitted  too  late  for  timely  program 
decisions.   Additionally,  data  submitted  during  developmental 
phases,  such  as  the  contractor  bi-weekly  or  bi-monthly  reports 
were  submitted  six  to  eight  weeks  after  being  generated,  often 
too  late  for  effective  utilization. 

The  problem  of  un-timely  data  occurs  for  several  reasons. 
One  reason  is  the  scope  and  complexity  of  required  tests  and 
data  acquisition  and  handling  problems.   A  second  is  the  lack 
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of  feedback  and  dialogue.   Another  is  the  existence  of  the 
review  chain,  a  necessary  evil  in  the  promulgation  of  a  T&E 
report.   It  is  the  authors  view  however  that  the  major  reason 
may  be  because  of  the  lack  of  understanding  or  acknowledgement 
of  the  role  that  the  DT&E  activities  play  in  the  overall  T&E 
process.   Each  output  report  becomes  a  cog  in  a  wheel  somewhere 
higher  up  the  T&E  organizational  hierarchy.   It  is  incumbent 
on  the  management  of  the  lower  systems  to  realize  that  they  play 
a  significant  part  (however  small)  in  the  overall  process  and 
their  system  outputs  must  be  timely  to  be  effective. 
3 .   System  Interfaces 

System  interfaces  occur  where  there  is  direct  inter- 
action between  two  or  more  different  systems  or  where  there  is 
an  information  flow  from  one  system  to  another.   As  expected, 
the  T&E  process  is  composed  of  many  interfaces.   There  are 
interfaces  between  DT&E,  OT&E,  and  PAT&E.   And  within  DT&E  there 
are  interfaces  between  the  T&E  activities,  the  contractors,  and 
NAVAIRSYSCOM.   For  instance,  T&E  of  the  F-14  was  composed  of 
elements  of  activities  such  as  NAVAIRTESTCEN ,  PACMISTESTCEN , 
and  the  contractors.   Two  major  problems  exist  in  this  area  of 
concern;  role  conflicts  between  T&E  systems  and  system  parochia- 
lism. 

Role  conflicts  exist  because  of  inadequately  established 
objectives  or  mission  statements  or  because  testing  of  a  weapon 
system  is  done  by  two  or  more  different  activities  separated 
by  large  geographic  distances.   Role  conflicts  arise  from  the 
historical  evolution  of  the  various  activities.   Each  activity 
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in  time  builds  up  an  expertise  based  on  what  their  overall 
mission  is  and  what  they  perceive  is  required  to  accomplish 
that  mission.   Thus  conflicts  arise  on  who  tests  what,  and  the 
results  is  usually  redundant  testing. 

In  the  weapon  system  area  of  the  F-14  program  this 
conflict  was  addressed  and  a  Joint  Evaluation  Team  (JET)  was 
formed  consisting  of  members  from  BIS,  NAVAIRTESTCEN ,  PACMIS- 
TESTCEN,  and  from  the  OT&E  community.   The  JET  proved  successful 
to  some  extent  (only  within  the  DT&E  community)  but  conflicts 
within  DT&E  activities  still  existed.   The  root  of  the  evil 
could  be  attributed  to  the  syndrome  known  as  parochialism. 

As  reported  by  Young  [Thesis,  1973]  the  self-serving 
parochial  attitude  sometimes  motivated  the  actions  of  organi- 
zations participating  in  the  F-14  T&E.   This  syndrome  is  diffi- 
cult to  document  and  validate,  and  some  feel  that  a  certain 
amount  of  parochialism  is  good,  that  it  creates  a  competitive 
spirit  among  T&E  systems.   But  the  important  point  to  remember 
about  parochialism  is  that  it  seems  to  breed  mutual  distrust 
between  organizations,  thus  curtailing  efficient  and  effective 
pursuit  of  overall  objectives.   Role  conflicts  and  parochial 
attitudes  are  caused  to  some  extent  by  a  lack  of  understanding 
of  "who  you  are" ,  what  function  do  you  play  in  the  overall  T&E 
process;  and  a  general  lack  of  concern  for  the  primary  objectives 

Interfaces  must  exist  for  the  T&E  process,  but  boundaries 
of  each  system  must  be  recognized  and  respected  to  minimize 
role  conflicts  and  parochialism  before  the  interfaces  can  be 
effective.   This  means  that  the  individual  system  objectives 
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must  be  established  as  to  purpose  and  scope  such  that  a  inter- 
related systematic  network  of  systems  can  be  established  with 
appropriate  interfaces. 

Interfaces  are  facilitated  by  the  well  known  "list  of 
key  personnel",  but  this  list  does  not  itself  establish  success- 
ful interaction  and  interface.   A  system  must  first  recognize 
that  interface  with  other  systems  is  necessary  to  meet  the  ob- 
jectives of  that  system  and  of  the  objectives  of  the  total 
system.   This  system  must  then  establish  interface  with  others, 
or  a  linking  within  the  T&E  system. 

4 .  System  Environment 

A  primary  function  of  the  system  is  to  recognize  that 
the  system  exists  within  an  environment.   This  environment 
changes,  either  dynamically  or  turbulently  in  nature  and  the 
system  must  react  and  adjust  to  survive.   For  the  DT&E  system 
the  environment  consists  of  such  things  as  top  management  policy, 
available  funds,  directives,  OT&E  requirements,  and  PAT&E  re- 
quirements to  mention  only  a  few.   The  manager  of  the  system 
guides  the  system  and  controls  its  destiny,  but  the  environment 
directs  and  controls  the  manager. 

5.  System  Effectiveness 

As  previously  stated,  system  effectiveness  is  a  measure 
of  how  well  the  objectives  of  the  system  have  been  met.   Thus 
an  effective  DT&E  system  is  one  which  performs  its  mission  and 
meets  its  objectives.   More  importantly  however  the  author  sug- 
gests that  an  effective  DT&E  system  is  one  which  meets  its 
objectives  in  the  context  of  the  total  T&E  process.   The  system 
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views  its  function  as  a  part  of  an  interrelated  group  of  T&E 
systems  performing  a  function  to  meet  the  overall  objectives 
of  the  T&E  process  of  the  weapon  system  in  concern.   An  import- 
ant factor  missing  in  this  context  is  an  evaluation  criteria. 
How  well  must  a  system  perform  or  meet  its  objectives  to  be 
effective? 

A  similiar  situation  exists  when  a  T&E  system  is  testing 
a  weapon  system.  Criteria  for  measuring  weapon  system  effective- 
ness are  supplied  in  the  form  of  specifications  and  guarantees. 
Note  however  that  the  fact  that  a  weapon  system  may  meet  all 
the  specifications  does  not  guarantee  an  effective  system.   This 
is  not  a  problem  of  objectives  not  being  stated;  but  is  a 
problem  of  the  objectives  being  erronously  stated  or  misinter- 
puted. 

It  is  well  recognized  that  system  effectiveness  in  the 
T&E  world,  whether  involving  a  management  or  a  weapons  system, 
is  not  easy  to  evaluate.   A  weapon  system  has  the  potential  of 
being  numerically  evaluated  thus  providing  some  measure  of 
effectiveness.   The  author  feels  however  that  the  evaluation 
of  a  management  system  is  a  subjective  factor,  one  that  requires 
rationalization  by  the  manager  and  a  view  of  the  whole  instead 
of  a  view  of  individual  parts. 

C.   OPERATIONAL  TEST  AND  EVALUATION 

The  functions  and  operations  of  contemporary  OT&E  have  just 
recently  evolved  (since  1971)  and  hence  are  short  on  tradition 
but  long  on  directive  policies.   The  OT&E  primary  function  of 
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"independent  evaluation"  has  been  both  praised  and  criticized 
hence  their  place  in  the  overall  T&E  process  is  worthy  of 
analysis.   Figure  6  will  aid  in  the  following  "systems  analysis" 
of  OT&E. 

1.  System  Objectives 

The  objectives  of  the  OT&E  system  are,  as  in  DT&E,  well 
defined  and  parallel  those  listed  under  paragraph  III-D-2.  Each 
activity  within  OT&E,  such  as  AIRTEVROR  FOUR  (VX-4) ,  have 
specific  objectives  such  as  to  perform  OT&E  on  fighter  aircraft 
systems.   Another  squadron,  VX-5,  performs  OT&E  on  attack  air- 
craft systems.   However,  one  of  the  deficiencies  within  the 
overall  T&E  process  as  reported  by  Bowes  [Thesis,  1974]  is 
the  "lack  of  adequate  OT&E  test  objectives".   This  problem 
still  exists  and  is  further  amplified  because  of  the  lack  of 
adequate  criteria  for  which  to  design  test  scenarios;  speci- 
fications and  guarantees  do  not  exist  for  combat  profiles  or 
for  operations  conducted  in  war  zones. 

2 .  System  Inputs/Outputs 

The  OT&E  community  has  much  the  same  problems  as  the 
DT&E  community  with  respect  to  funds,  personnel,  airplanes, 
and  reports.   Peculiar  problem  areas  to  OT&E  however  lie  in 
the  area  of  airplanes  and  personnel. 

Being  operationally  oriented  necessitates  the  use  of 
production  configured  airplanes.   This  unfortunately  is  not 
always  possible  in  the  T&E  process  until  far  down  the  develop- 
ment cycle.   Hence  a  true  operational  assessment  of  a  weapon 
system  cannot  be  made  until  dangerously  close  to  the  milestone 
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for  major  production  decision.   The  role  and  objectives  of  OT&E 
are  thus  difficult  to  achieve  in  a  strict  sense.   Of  necessity, 
flying  has  to  be  performed  on  prototype  airplanes  with  systems 
possibly  still  in  development.   The  final  production  configured 
airplane  may  not  be  the  same  as  tested  by  OT&E. 

Lack  of  analytical  and  technical  personnel  is  another 
factor  adversly  affecting  the  function  of  OT&E.   Program  planning 
and  data  analysis  support  is  currently  being  performed  by 
nbn-airframe  type  contractors  working  under  the  control  of 
OPTEVFOR.   This  is  expensive  and  undersirable  in  some  respects. 
These  functions  could  just  as  well  be  performed  by  civil  service 
engineers  with  the  appropriate  expertise. 
3 .   System  Interfaces 

Interfaces  within  the  OT&E  system  are  primarily  with 
the  squadrons,  Headquarters,  DT&E  activities,  and  the  contractors 
who  perform  data  analysis.   An  apparent  interface  problem,  as 
perceived  by  this  author,  may  lie  within  the  interface  of 
Headquarters  and  the  squadron  level.   This  is  a  result  of 
possible  differences  of  opinion  as  to  the  interpretation  of 
the  stated  objectives.   An  educational  and/or  communicational 
in-house  process  can  rectify  the  situation. 

A  unique  interface  problem  facing  OT&E  is  the  dilemma 
of  when  to  get  involved  in  the  T&E  of  the  Weapon  system.   Too 
early  an  entry  may  mean  disruption  of  the  developmental  process 
and  may  involve  a  prototype  airplane  non-representative  of  the 
production  lot.   Too  late  an   entry  and  OT&E  will  have  little 
or  no  input  on  the  major  production  decision  and  suitability 
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for  fleet  operation.   The  author  perceives  internal  conflict 
within  DoD  on  the  exact  role  of  OT&E  and  further  still,  the 
author  perceives  conflict  within  the  OT&E  community  itself  on 
what  it's  role  is  to  be. 

4 .  System  Environment 

The  environment  that  OT&E  operates  within  is  much  like 
that  of  DT&E.   The  major  factors  being  CNO  policy/directives, 
DT&E,  PAT&E,  funds,  and  test  aircraft  availability.   The  OT&E 
system  too  must  recognize  that  the  environment  exists  and  is 
ever  changing  in  order  to  survive. 

5 .  System  Effectiveness 

As  with  the  DT&E  system,  the  effectiveness  of  the  OT&E 
system  is  a  function  of  how  well  the  system  meets  its  objectives 
and  how  well  it  fits  into  the  interrelated  group  of  other  T&E 
elements.   Managerial  effectiveness  can  be  attained  by  recogni- 
zing this,  just  as  with  any  other  T&E  activity. 

The  major  problem  lies  in  the  lack  of  test  objectives 
and  criteria  for  determing  the  effectiveness  of  the  weapon 
systems  themselves  when  evaluating  in  an  operational  environ- 
ment.  A  test  program  should  be  designed  to  measure  the  suit- 
ability or  effectiveness  of  a  system  against  some  criteria 
which  may  either  be  supplied  by  an  Operational  Requirement  or 
Detailed  Specification.   An  operational  effectiveness  criteria 
is  needed  for  which  the  OT&E  system  can  evaluate  the  results 
of  their  tests. 

Thus  the  objectives  of  OT&E  are  stated,  but  the  missing 
ingredient  is  a  test  criteria.   The  criteria  of  "mission 
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suitability"  is  well  and  good  but  different  pilots  have  differ- 
ent views  as  to  what  is  suitable,  and  more  importantly,  different 
abilities  to  evaluate  the  system.   By  the  "seat  of  the  pants" 
is  not  good  enough,  hence  the  effectiveness  of  the  weapon 
system  is  at  best  a  subjective . evaluation.   This  problem  is 
however  not  unique  to  OT&E.   The  DT&E  community  is  hard  pressed 
to  evaluate  system  effectiveness  as  previously  discussed.  Meeting 
or  exceeding  a  specification  number  does  not  make  the  system 
effective.   Apparently  the  weapons  planners  do  not  know  enough 
yet  about  threat  forecasting  and  combat  effectiveness  criteria 
to  tie  a  number  to  it,  and  under standly  so;  the  environment 
for  which  the  system  is  designed  changes  constantly. 

D.   PRODUCTION  ACCEPTANCE  TEST  AND  EVALUATION 

The  functions  and  objectives  of  PAT&E  have  evolved  along 
with  DT&E  throughout  the  years  and  are  carried  out  by  the  BIS. 
Figure  7  will  aid  in  the  analysis  of  the  PAT&E  system. 
1.   System  Objectives 

The  objectives  of  PAT&E  as  carried  out  by  BIS  are  those 
listed  under  paragraph  III-D-3  and  need  not  be  reiterated  here. 
Primarily  the  role  of  BIS  is  that  of  recommending  conditions 
of  acceptance  or  rejection  for  service  use  of  a  weapon  system. 
This  is  done  by  providing  timely  inputs  of  technical  aspects 
of  the  weapon  system  to  SecNav  via  CNO.   These  objectives  have 
been  with  the  total  T&E  process  for  some  time  and  tend  to  assure 
an  overall  satisfactory  weapon  system  for  the  fleet.   Additionally, 
test  objectives  exist  in  the  form  of  specifications  and  guaran- 
tees for  which  the  weapon  system  must  meet. 
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ENVIRONMENT 


FIGURE  7 

MODEL  OF  THE  PRODUCTION  ACCEPTANCE 
TEST  AND  EVALUATION  SYSTEM 
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2.  System  Inputs/Outputs 

Inputs  for  the  PAT&E  system  include  airplanes,  funds, 
and  T&E  activities.   The  BIS  tasks  appropriate  T&E  activities 
to  determine  compliance  with  weapon  system  requirements.   Per- 
sonnel from  these  T&E  activities  in  effect  become  a  member  of 
the  "BIS  team"  and  actually  work  for  the  Board.   Funds  and  air- 
craft are  requested  from  NAVAIRSYSCOM  by  BIS  who  then  gains 
control  of  these  assets. 

The  outputs  from  BIS  consists  primarily  of  individual 
"Yellow-Sheets"  which  describe  specification  and/or  mission 
deficiencies  of  the  weapon  system  and  must  be  supported  by 
substantiating  data.   These  "Yellow-Sheets"  are  followed  up 
with  a  formal  all  inclusive  report  which  is  endorsed  through 
the  hierarchy  to  the  SecNav. 

3 .  System  Interfaces 

Interfaces  within  the  PAT&E  system  are  much  the  same 
as  in  the  DT&E  system  with  the  exception  that  the  "Board"  is 
a  primary  elemental  organization.   Note  that  BIS  itself  is  not 
a  testing  agency  or  testing  complex  per  se,  it  is  a  reporting 
agency.   The  existing  T&E  activities  and  facilities  do  the  work 
for  BIS  as  BIS  members  and  report  test  results  in  required 
form  to  the  President  of  the  Board  who  in  turn  reports  to  SecNav 
via  the  CNO. 

4 .  System  Environment 

The  environment  of  PAT&E  is  much  like  that  of  DT&E  and 
OT&E.   The  primary  factors  being  CNO  policy,  funds,  test  air- 
planes, and  the  individual  T&E  activities. 
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5.   System  Effectiveness 

Effectiveness  of  the  PAT&E  system  can  be  evaluated  in 
terms  of  how  well  it  meet's  its  objectives  previously  discussed. 
The  problem  is  that  no  one  knows  the  deficiencies  and/or  prob- 
lems that  are  not  discovered,  only  those  that  are  discovered. 
Hence  the  effectiveness  of  the  PAT&E   system  may  be  a  measure 
ultimately  of  successful  fleet  operation.   If  the  weapon  system 
functions  well,  then  the  entire  T&E  process  has  been  performed 
well.   If  deficiencies  are  discovered  by  the  fleet  during 
operations  that  something  faultered  during  the  T&E  cycle.   Of 
course  these  are  again  subjective  areas  of  discussion  and  again 
the  point  of  evaluation  effectiveness  of  a  T&E  system  is  a 
difficult  one  to  attempt. 

E.   SUMMARY  OF  THE  TYPES  OF  TEST  AND  EVALUATION 

The  preceeding  sections  have  presented  a  systems  view  of 
the  three  types  of  T&E;  DT&E,  OT&Ef  and  PAT&E,  each  of  which 
have  established  objectives.   However  these  three  systems  are 
comprised  themselves  of  many  lower  order  systems  which  in  many 
cases  may  have  objectives  not  in  congruence  with  the  larger 
systems  objectives.   The  salient  point  is  however  that  each 
system  can  be  analyzed  in  terms  of  typical  systematic  charac- 
terisitcs  of  the  dynamic  systems  model,  i.e.  objectives,  inputs 
and  outputs,  interfaces,  environment,  and  effectiveness. 
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V.   A  SYSTEMS  LOOK  AT  THE  TOTAL  T&E  PROCESS 

The  three  types  of  T&E  have  been  analyzed  using  a  systems 
approach.   Each  individual  system  has  its  fundamental  elements 
and  systematic  characteristics.   But  the  T&E  process  must  be 
viewed  as  a  whole  instead  of  as  individual  parts.   The  systems 
of  DT&E,  OT&E,  and  PAT&E  are  all  interrelated  elemental  systems 
of  the  whole  T&E  system  process. 

The  T&E  process  within  the  Navy  is  currently  in  a  flexing 
posture.   New  policies  and  trends  coupled  with  stricter  control 
over  scarce  resourses  triggered  by  the  economical  and  political 
environments  are  impacting  T&E. significantly.   Rapidly  advancing 
technology  in  both  the  weapon  systems  themselves  and  the  media 
and  methods  for  conducting  T&E  are  adding  to  the  complexities 
of  T&E  today.   A  systems  look  at  the  total  T&E  process  as  it 
exists  now  is  warranted,  for  it  illustrates  several  significant 
problems  and  areas  of  conflict.   The  recent  experience  on  the 
F-14  weapon  system  and  some  of  the  experience  to  date  of  the 
F-18  program  offer  evidence  that  some  systematic  approach  is 
indeed  needed.   Figure  8  is  presented  to  depict  the  current 
environment  and  turmoil  within  the  T&E  system  and  to  aid  in 
understanding  the  succeeding  paragraphs.   The  objectives, 
interfaces,  and  effectiveness  features  of  a  dynamic  system  are 
again  used  to  analyze  the  total  T&E  process.   The  input/output 
and  environmental  dimensions  are  not  presented  here  because  they 
expand  the  systems  study  to  the  Congressional  level  and  impose 
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FIGURE  8 


MODEL  OF  THE  TOTAL  TEST  AND  EVALUATION   SYSTEM 
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correspondingly  inuneasurables  upon  the  study  such  as  bureau- 
cratic politics.   This  expansion,  although  certainly  valid,  is 
not  needed  to  put  forth  this  thesis. 

A.   SYSTEM  OBJECTIVES  OF  T&E 

The  objectives  of  DT&E,  OT&E,and  PAT&E  have  been  listed 
and  discussed  in  the  preceeding  chapter.   This  section  presents 
the  objectives  of  the  total  T&E  process.   In  simplistic  and 
condensed  form,  the  primary  objective  of  T&E  is  to  perform 
tasks  and  evaluate  test  results  which  are  conducted  to: 

(1)  Support  design  and  development  activities. 

(2)  Measure  performance  against  specified  acceptance 
criteria. 

(3)  Confirm  operational  effectiveness  and  suitability. 

(4)  Ensure  satisfactory  operation  with  related  inter- 
facing systems. 

(5)  Validate  the  adequacy  of  documentation  for  support 
and  test  equipment,  personnel  training;  maintenance 
and  operation  of  the  whole  system/subsystem  and  other 
elements . 

Note  that  objective  (1)  corresponds  to  DT&E  functions; 
objective  (2)  to  DT&E  and  PAT&E;  and  objective  (3)  to  OT&E 
and  PAT&E.   Objectives  (4)  and  (5)  are  unfortunately  sometimes 
not  addressed,  either  because  of  ignorance  or  of  constraints 
on  the  total  flight  test  program  itself  such  as  lack  of  a  fully 
configured  aircraft.   Thus  the  overall  objectives  of  T&E  are 
separated  into  the  three  types  of  T&E  systems.   The  subsequent 
integration  of  these  objectives  toward  the  whole  system  objec- 
tives are  sometimes  overlooked. 
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B.   SYSTEM  INTERFACES  WITHIN  T&E 

The  elemental  systems  comprising  the  T&E  system  which 
interface  with  each  other  consist  of  the  DT&E,  OT&E,  and  PAT&E 
systems  as  presented  in  the  previous  chapter.   As  evidenced  in 
Figure  8  these  systems  are,  unfortunately,  not  always  inter- 
related into  a  cohesive  whole  necessary  for  effective  T&E.  Role 
conflicts  exist  between  the  systems  and  some  guidance,  direction 
and/or  framework  for  thinking  must  be  implanted  in  the  T&E 
process  to  minimize  this  inefficient  use  of  scarce  resourses. 

Several  areas  of  conflict  within  the  F-14  T&E  program  were 
documented  by  Young  [Thesis,  19  73]  and  are  summarised  here. 
One  area  of  conflict  is  between  the  DT&E  and  OT&E  systems.   The 
crux  of  the  problem  lies  within  the  question  of  when  should 
OT&E  become  involved  in  the  overall  T&E  process,  if  at  all. 
One  argument  says  that  little  OT&E  is  needed  since  the  DT&E 
community  is  able  to  conduct  tests  with  an  objective,  inde- 
pendent approach  by  operationally  experienced  flight  crews, 
just  as  OT&E  does.   There  are  fears  that  OT&E  can  disrupt  the 
DT&E  process  with  too  early  an  involvement  and  also  be  counter- 
productive to  the  OT&E  objectives  because  the  airplane  systems 
configurations  are  usually  not  stabilized  until  far  downstream 
in  the  T&E  process. 

The  OT&E  community  feel,  of  course,  that  their  involvement 
in  the  T&E  process  is  necessary,  for  that  is  their  purpose  in 
being.   The  primary  concern  of  OT&E  is  apparently  not  one  of 
early  involvement,  but  one  of  adequate  and  timely  involvement 
prior  to  the  first  major  production  decision  milestone.   Too 
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meet  their  objectives  the  OT&E  phase  needs  calender  time  within 
the  DT&E  process  and  dedicated  test  airplanes  configured  as 
close  as  possible  to  production  configurations. 

Another  area  of  conflict  is  between  PAT&E  and  OT&E.   Both 
systems  report  to  CNO  on  the  adequacy  of  the  total  weapon 
system.   The  OT&E  system  reports  on  "adequacy  of  service 
suitability"  while  PAT&E  (BIS)  reports  on  "acceptance  for 
service  use" .   Even  though  the  OT&E  system  reports  on  opera- 
tional effectiveness  and  PAT&E  reports  on  technical  effective- 
ness both  realistically  offer  the  objective  opinion  of  whether 
the  system  is  ready  for  the  fleet.   However  the  OT&E  system 
reports  directly  to  CNO  throughout  the  T&E  process  while  PAT&E 
does  not  report  until  further  downstream  the  T&E  process.   The 
OT&E  system  is  thus  visible  to  CNO  somewhat  before  the  PAT&E 
system.   The  F-18  program  currently  has  BIS  scheduled  after  the 
first  major  production  decision  milestone,  certainly  contra- 
dictory to  the  objectives  of  PAT&E.   The  BIS  recognizes  that 
the  PAT&E  system  has  little  effect  on  the  major  production 
decision  process  if  the  T&E  system  exists  as  described.   Hence 
BIS  needs  an  "initial"  look  at  the  weapon  system  in  order  to 
function  consistently  within  their  stated  objectives.   This 
"Initial  Trials  Phase"  has  long  been  in  existence  and  should 
remain  actively  so  such  as  to  allow  BIS  timely  meeting  of  its 
objectives. 

C.   SYSTEM  EFFECTIVENESS 

As  previously  discussed,  effectiveness  is  a  subjective 
oarameter  in  the  T&E  world.   The  individual  T&E  subsystems, 
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(DT&E,  OT&E,  and  PAT&E)  strive  to  meet  the  stated  objectives  of 
their  respective  systems  and  a  subjective  statement  as  to  the 
adequacy  of  meeting  those  objectives  can  be  made.   Each  system 
acts  within  its  own  environment  and  functions  accordingly. 

The  effectiveness  of  the  total  T&E  system  is  just  as 
illusive  as  that  of  the  individual  systems  comprising  T&E. 
The  effectiveness  of  the  T&E  system  as  a  cohesive  whole  is 
in  question.   Evidence  to  date  indicates  that  the  T&E  system 
is  low  in  effectiveness  because  of  existing  conflicts  which 
tend  to  detract  from  the  overall  goals  and  waste  significant 
amounts  of  scarce  resourses.   The  F-14  JET  concept  was  an 
example.   The  JET  was  formed  to  integrate  all  testing  require- 
ments.  All  three  T&E  systems  were  involved  but  only  the  inter- 
face of  DT&E/PAT&E  was  fluid  (and  only  to  a  minimal  degree) . 
The  objectives  of  the  JET,  as  viewed  by  DT&E,  were  to  satisfy 
the  BIS  requirements.   The  means  to  the  end  were  however  in^ 
terpretated  differently  by  the  various  individuals  within  the 
DT&E  system.   The  OT&E  system,  even  though  involved  "on  paper" 
within  the  process,  was  not  readily  accepted.   Hence  their 
objectives  fell  by  the  wayside.   Thus  the  objectives  of  the 
JET  concept,  to  unite  the  T&E  systems,  failed. 

There  are  some  indications  that  the  future  F-18  program  is 
encountering  conflicts  within  the  total  T&E  system.   Of  signi- 
ficance is  the  situation  within  DT&E  and  the  T&E  activity  sub- 
systems where  the  missile  expertise  of  the  weapon  system  are 
not  currently  actively  participating  within  the  DT&E  process. 
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Conflict  is  good,  too  a  degree.   Without  conflict  certain 
points  of  view  and  alternatives  would  not  be  fostered.   However 
the  result  of  conflict  within  the  T&E  process  is  rapid,  needless 
expenditure  of  the  most  significant  resourse  for  T&E,  namely 
flight  test  time.   Conflict  must  be  controlled  so  that  the 
total  objectives  of  the  T&E  system  are  visible,  and  each  of 
the  subsystems  within  the  T&E  system  align  its  respective  ob- 
jectives to  the  primay  objective  of  the  total  system.   The 
T&E  system  must  be  composed  of  interrelated  parts  acting  as  a 
cohesive  whole  following  a  given  logical  sequence.   The  three 
systems  are  interdependent  on  each  other  for  effective  overall 
T&E  of  a  weapon  system. 

D.   SUMMARY  OF  THE  TOTAL  T&E  PROCESS 

The  total  T&E  system  process  is  comprised  of  the  DT&E,  OT&E, 
and  PAT&E  systems.   Even  though  the  systems  have  complementary 
stated  objectives  (in  directive  form)  their  actual  behavior 
is  such  that  one  might  conclude  that  the  objectives  of  these 
systems  are  not  really  complementary  and  do  not  act  in  total 
alignment  with  the  objectives  of  the  T&E  process.   Role  conflict 
and  parochialism  exist  and  each  system  must  recognize  its  part 
in  relation  to  the  whole  before  an  effective  T&E  system  can 
exist. 
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VI.   THE  SYSTEMS  APPROACH  TO  MANAGEMENT  AND  THE  T&E  PROCESS 

Thus  far  this  thesis  has  established  that  the  T&E  process 
is  in  fact  composed  of  three  large  systems  interacting  to  per- 
form the  test  and  evaluation  function.   These  larger  systems, 
the  DT&E,  the  OT&E,  and  PAT&E  systems  are  themselves  composed 
of  many  lower  order  systems.   It  has  been  shown  that  all  these 
systems,  including  the  T&E  system  itself,  displays  the  systemic 
characteristics  of  a  dynamic  system.   The  concept  of  the  systems 
approach  to  management  has  been  put  forth  and  the  importance  of 
viewing  the  system  as  a  whole  has  been  stressed.   As  previously 
pointed  out,  before  the  manager  of  a  T&E  system  can  effectively 
manage  he  must  know  the  process  of  his  system  and  its  role  in 
the  overall  T&E  system. 

A.   MANAGEMENT  AND  THE  TEST  AND  EVALUATION  PROCESS 

The  management  functions  within  the  T&E  process  can  be 
better  understood  if  the  T&E  system  is  further  analyzed. 
Chapters  III  through  IV  have  discussed  the  types  of  T&E  from 
a  systems  view.   T&E  can  also  be  analyzed  by  looking  at  its 
various  phases.   As  outlined  by  Perry  [1976]  in  discussing  the 
role  of  T&E,  the  T&E  process  consists  of  five  basic  phases: 

(1)  Planning 

(2)  Programming 

(3)  Budgeting 

(4)  Testing 

(5)  Evaluation 
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A  good  manager  relies  on  ideas,  resourses,  things,  tools, 
and  people  to  achieve  objectives.   He  utilizes  such  managerial 
functions  as  planning,  organizing,  staffing,  directing,  and 
controlling  as  prescribed  by  such  authors  as  Koontz   and 
O'Donnell  [1968] .   More  importantly  these  functions  must  be 
harmonized  and  put  in  juxtaposition  with  the  five  basic  phases 
of  T&E.   This  correlation  is  depicted  in  Figure  9  which  shows 
the  relationship  of  basic  managerial  functions  to  the  basic 
phases  of  the  T&E  process.   Perry  describes  the  basic  phases 
of  T&E  in  the  following  manner. 

During  the  planning  phase  the  role  of  test  and  evaluation 
is  to  work  for,  work,  with,  and  provide  feedback  to  the  planning 
arm  of  the  organization.   As  such,  seven  steps  are  required 
of  the  manager  for  the  planning  function: 

(1)  Understand  the  need  of  the  program  manager. 

(2)  Set  test  and  evaluation  objectives  in  harmony  with 
the  program  objectives. 

(3)  Develop  time  phasing  for  entire  life  cycle  of  the 
program. 

(4)  Establish  test  program  goals  and  milestones. 

(5)  Select  appropriate  procedures  and  methods  to  be 
applied  during  T&E. 

(6)  Construct  a  test  and  evaluation  master  plan  which  is 
documented  transmittable  to  other  organizations  as 

a  TEMP. 

(7)  Share  with  other  managers  in  formulating  the  strategies, 
tactics  and  policy  to  achieve  success  of  the  program. 

The  programming  phase  is  concerned  with  a  quantified  defi- 
nition of  what  work  has  to  be  done  and  what  work  resourses 


65 


: 


T&E  ACTIVITIES 

1.    PLANNING 

UNDERSTAND  NEED 

SET  OBJECTIVES 

TIME  PHASING 

GOALS  AND  MILESTONES 

SELECT  METHODS 

CONSTRUCT  TEMP 

DEVELOPPOLICY 


PROGRAMMING 
ESTABLISH  T&E  STRUCTURE 
DELINEATE  WORK  RELATIONS 
PRICE  WORK  RESOURCES 
CHOOSE  BEST  PROGRAM 


3.  BUDGETING 
ALLOCATE  QUALITY 
DETERMINE  QUANTITY 
APPORTION  TIME 
BUDGET  MONEY 

4.  TESTING 

FORMULATE  CRITERIA 
DESIGN  THE  EXPERIMENT 
FIRM  THE  TEST  MODEL 
CONDUCT  TESTS 
ANALYZE  DATA 

5.  EVALUATION 

SYNTHESIZE  DATA 
COMPARE  TO  CRITERIA 
DERIVE  RESULTS 
REPORT 
SHARE  IN  RISK 


FIGURE  9 

RELATIONSHIP  OF  MANAGERIAL 
FUNCTIONS  TO  THE  PHASES  OF  TEST  AND  EVALUATION 
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will  be  necessary  to  accomplish  a  viable  test  and  evaluation 
program.   The  basic  managerial  functions  of  organizing  and 
staffing  are  important  for  this  phase;  as  such  there  are  four 
basic  steps: 

(1)  Establish  the  test  and  evaluation  work  structure  by 
making  sure  that  all  the  system  elements  (both  verti- 
cal and  horizontal)  are  properly  put  in  perspective 
as  to  what  tests  and  what  evaluations  must  be  per- 
formed. 

(2)  Delineate  the  work  relationship  by  identifying  how 
the  human  element  and  the  organizations  (internally 
and  externally)  will  perform  the  structured  work. 

(3)  Price  the  work  resourses  by  estimating  the  costs  and 
benefits  to   be  provided  by  test  and  evaluation. 

(4)  Choose  the  best  program  alternative  and  provide  a 
feedback  to  the  program  manager  for  his  acceptance. 

The  budgeting  phase  follows  planning  and  programming.   The 
test  and  evaluation  organization  must  be  concerned  with  the 
quality  level  of  the  product ,  the  quantity  of  T&E  information 
to  be  generated,  the  quantity  of  resourses  to  be  applied  in 
reaching  the  desired  level  of  confidence  regarding  critical 
unknowns   and  these  into  explicit  schedules.   This  permits  the 
monetary  and  financial  budgeting  requirements  for  the  total 
T&E  program.   As  such,  four  basic  steps  concerned  with  alloca- 
tion of  the  budget  are: 

(1)  Quality 

(2)  Quantity 

(3)  Time 

(4)  Money 
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The  pattern  of  the  testing  and  the  evaluation  functions 
are  derived  from  the  scientific  approach  of  management.  During 
the  testing  phase  five  activities  are  required: 

(1)  Formulate  the  evaluation  criteria. 

(2)  Design  the  experiments. 

(3)  Identify  exactly  how  elements  of  hardware,  software, 
and  "brainware"  will  be  integrated  during  the  test 
program. 

(4)  Conduct  the  tests. 

(5)  Analyze  and  process  the  required  data. 

The  evaluation  phase  continues  the  testing  process  by 
accomplishing  five  more  steps: 

(1)  Synthesize  the  data  obtained  from  testing  by  making 
sure  that  the  information  is  relevant,  germaine, 
and  valid. 

(2)  Compare  the  synthesized  data  with  performance  stan- 
dards and  the  evaluation  criteria. 

(3)  Derive  the  results  required  by  management. 

(4)  Report  the  results  in  language  useable  and  under- 
standable by  management. 

(5)  Provide  recommendations  as  to  confidence  and  overall 
risk  of  the  system  or  product. 

Collectively  this  is  a  closed  loop  process  which  relies  on 
ideas,  resourses,  things,  tools,  and  people.   Such  a  test  and 
evaluation  process  model  can  serve  the  T&E  manager  as  a  check- 
list of  systematic  and  sequential  steps  within  his  organization. 
In  the  T&E  system  the  manager  must  recognize  that  the  objectives 
to  meet  through  use  of  resourses,  ideas,  and  people  are  the 
ones  consistent  with  the  "whole"  T&E  process,  which  itself 


68 


consists  of  the  basic  phases  of  planning,  programming,  bud- 
geting, test,  and  evaluation.   The  manager  must  also  not 
loose  site  of  the  fact  that  the  T&E  system  is  merely  a  sub- 
system itself.   T&E  is  a  managerial  tool  to  be  used  to  provide 
information  necessary  to  make  development,  procurement,  and 
deployment  decisions. 

B.   STRATEGY  FORMULATION  -  IMPLEMENTATION  OF  THE  SYSTEMS  APPROACH 

The  task  of  how  to  implement  the  systems  approach  to  manage- 
ment becomes  an  important  issue,  for  merely  talking  about  the 
concepts  and  methodology  will  not  create  action.   This  approach, 
once  its  validity  and  usefulness  have  been  recognized,  must  be 
implemented.   Implementation  can  be  made  to  any  system  for  each 
system  is  independent  as  to  "how"  it  is  managed.   Systems  are 
not  independent  as  to  "how"  they  function,  for  each  has  inter- 
related elements  forming  a  whole.  A  lesson  from  the  corporation 
world  on  the  concepts  of  strategy  is  worthy  of  note. 

The  systems  approach  to  management  may  be  initiated  by 
applying  the  concepts  of  strategy  formulation  practiced  by 
many  corporations.   Corporate  strategy  can  best  be  defined, 
according  to  Uyterhoeven  [1973]  by  looking  at  the  purposes 
it  serves;  to  provide  both  direction  and  cohesion  to  the  enter- 
prise.  Providing  direction  is  the  traditional  objective  of 
corporate  strategy:   to  give  the  company  a  sense  of  purpose 
and  mission.   Providing  cohesion  is  however  more  essential  than 
providing  direction.   In  providing  cohesion,  corporate  strategy 
influences  the  unit  strategies  rather  than  permitting  the 
latter  to  shape  the  corporate  response.   Furthermore,  to 
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establish  cohesion  corporate  strategy  must  establish  priorities 
among  the  units.   Finally  cohesion  requires  that  the  activities 
of  the  various  units  be  interrelated.   This  integration  aspect 
not  only  provides  a  common  sense  of  direction  to  the  units  but 
also  permits  the  total  organizational  response  to  be  more  ef- 
fective than  that  of  the  sum  of  its  parts.   These  concepts 
from  Uyterhoeven  about  corporate  strategy  are  readily  applicable 
to  any  T&E  organizational  system.   The  model  developed  by 
Uyterhoeven  to  formulate  strategy  is  presented  below  with 
amplifying  statements  showing  the  relationship  to  the  T&E 
process.   The  formulation  of  corporate  strategy  encompasses 
seven  steps: 

(1)  Develop  a  Strategic  Profile  which  concerns  itself 
with  the  business  or  objectives  of  the  system. 

(2)  Evaluate  the  Environmental  Dimension,  including 
such  factors  as  politics,  economics,  and  technology. 

(3)  Arrive  at  a  Strategic  Forecast  which  relys  on  the 
validity  of  the  environmental  analysis.   This  concerns 
itself  with  predicting  the  future  shape  or  posture 

of  the  environmental  dimension. 

(4)  Perform  a  Resourse  Audit  which  focuses  on  the  internal 
dimensions.   Resourses  are  identified  and  evaluated 
and  the  strengths  and  weaknesses  of  the  organization 
are  developed. 

(5)  Develop  Strategic  Alternatives  which  concerns  itself 
with  how  can  the  organization  respond  to  a  given 
stimuli,  what  are  the  range  of  choices  for  meeting 
an  obligation  or  objective. 

(6)  Perform  a  Test  of  Consistency  which  relates  what  the 
system  or  organization  is  "able"  to  do  with  respect 
to  its  available  resourses  to  what  is  "possible"  in 
its  given  external  environment. 

(7)  The  manager  makes  a  Strategic  Choice  of  the  available 
alternatives  which  provide  consistency  within  the 
stated  goals  or  objectives. 
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The  author  must  however  caution  the  readers  not  to  over- 
simplify the  issues  just  discussed.   Implementation  of  a 
"corporate  strategy"  which  in  turn  initiates  the  application 
of  the  systems  approach  to  management  may  meet  unsurmountable 
obstacles.   These  implementation  problems  are  discussed  by 
Allison  [1971]  in  his  analysis  of  the  "Cuban  missile  crisis". 
In  essence,  the  rational  alternatives  and  choices  arrived  at 
may  be  strongly  affected  by  organizational  policies  and  bureau- 
cratic politics,  either  adversely  or  reinforcingly ,  depending 
on  the  situation  and  the  people  involved. 

C .   SUMMARY 

The  systems  approach  provides  the  manager  with  a  framework 
for  thinking  about  the  job  of  management.   Coupling  this  approach 
with  the  systematic  nature  of  the  T&E  process  offers  an  ideal 
tool  for  the  manager  to  use.   The  classic  functions  of  manage- 
ment can  be  kept  in  juxtaposition  with  the  phases  of  T&E  to 
provide  a  better  insight  to  management  within  the  T&E  process. 
Implementation  of  a  strategy  is  perhaps  the  most  important 
first  step  in  practically  using  the  systems  approach  and  as 
such  stimulates  the  manager  to  think  in  the  proper  perspective. 
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VII.   CONCLUSIONS 

This  thesis  has  developed  two  basic  conclusions  as  a  result 
of  utilizing  a  dynamic  systems  model  to  analyze  the  Test  and 
Evaluation  process.   The  first  is  that  the  Test  and  Evaluation 
process  can  be  represented  as  a  conglomerate  of  lower-order 
systems.   Each  layer  of  systems  can  be  characteristically 
analyzed  by  focusing  on  the  objectives,  input/outputs,  inter- 
faces, environmental,  and  effectiveness  dimensions. 

The  systematic  characteristics  of  the  Test  and  Evaluation 
process  suggests  that  a  systematic  approach  to  management  can 
be  readily  applied  by  managers.   This  thus  constitutes  the 
second  basic  conclusion  of  this  thesis.   A  paradigm  for  the 
Test  and  Evaluation  process,  useful  in  many  respects,  is  the 
systems  approach  to  management.   This  approach  provides  the 
managers  a  systematic  approach  to  the  job  of  management,  one 
which  the  author  finds  appealing  to  the  engineering  background. 
This  framework  of  thinking  guides  the  manager  to  view  the 
whole  instead  of  only  his  elemental  part.   More  importantly 
the  manager  will  recognize  the  importance  of  properly  aligned 
objectives.   The  formulation  of  a  strategy  becomes  important 
in  implementation  of  this  paradigm  for  it  provides  the  means 
to  initiate  this  approach  to  management  and  it  provides  the 
groundwork  for  direction  and  cohesion  so  needed  by  the  system. 


72 


APPENDIX  A 
ORGANIZATIONS  WITH  WHICH  INTERVIEWS  WERE  CONDUCTED 

Developmental  Test  and  Evaluation 

1.  Naval  Air  Test  Center,  Patuxent  River,  Md. 

2.  Pacific  Missile  Test  Center,  Pt  Mugu,  Calif. 

3.  Air  Force  Flight  Test  Center,  Edwards  AFB,  Calif. 

4.  Naval  Air  Systems  Command,  AIR  5102  Office,  Wash.  D.C. 

Operational  Test  and  Evaluation 

5.  AIRTEVRON  FOUR  (VX-4),  Pt  Mugu,  Calif. 

Production  Acceptance  Test  and  Evaluation 

6.  Sub  Board  of  Inspection  and  Survey,  Patuxent  River,  Md 
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APPENDIX  B 
EVOLUTION  OF  THE  VARIOUS  MANAGEMENT  THEORIES 

A.   INTRODUCTION 

The  systems  approach  to  management  is  one  of  the  many 
theories  of  management  that  exist  today.   There  are  however 
some  clearly  identifiable  concepts  which  provide  a  systematic 
look  at  these  various  theories.   The  following  paragraphs 
[Luthans,  1973]  and  Figure  10  provide  an  aid  in  understanding 
these  theories  and  their  relationship  to  each  other. 

The  starting  point  of  management  theories  can  be  traced  to 
Henri  Fayol.   In  1916  he  identified  the  universal  functions 
of  management  as  processes  of  planning,  organizing,  commanding, 
coordinating,  and  controlling.   Fayol 's  work  did  not  however 
become  the  mainstream  of  management  theory  until  the  1950' s. 
His  process  theories  have  remained  relatively  intact,  however 
some  of  the  terminology  has  changed.   Fayol 's  "commanding" 
process  is  now  known  as  "directing".   Dispite  the  changes  in 
terminology  and  principles  the  universality  assumption  of  the 
process  approach  is  still  made  and  remains  as  the  theoretical 
base  for  management. 

In  recent  years  the  process  approach  has  been  overrun  by 
other  theoretical  approaches.   By  1960  two  separate  paths 
emerged  from  the  process  approach  in  opposite  directions.   These 
new  paths  became  known  as  the  quantitative  and  behavior  ap- 
proaches to  management. 
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EVOLUTION  OF  THE  VARIOUS  MANAGEMENT  THEORIES 
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B.   APPROACHES  TO  MANAGEMENT 

1-   The  Quantitative  Approach 

The  quantitative  approach,  emerging  in  early  1960,  has 
its  roots  in  the  scientific  management  movement  that  actually 
predates  the  process  approach.   This  approach  was  characterized 
by  the  techniques  of  operations  research  which  used  various 
mathematical  models  to  solve  decision  problems.   However  it 
soon  became  apparent  that  this  approach  fell  short  of  providing 
a  theoretical  base  for  management  as  a  whole,  although  opera- 
tions research  techniques  did  provide  effective  tools  for 
management  decision  making. 

Starting  in  about  1970,  the  quantitative  approach 
turned  away  from  emphasis  on  narrow  operation  research  tech- 
niques toward  a  broader  perspective  of  management  science. 
In  addition  to  the  quantitative  decision  making  techniques 
and  model  building  as  in  OR  the  function  of  computerized  in- 
formation systems  and  operations  management  were  incorporated. 
This  then  marked  the  return  toward  a  more  broadly  based  theory 
of  management. 

2.   The  Behavioral  Approach 

The  other  separation  from  the  process  approach,  the 
behavior  approach,  was  charactirized  by  human  relations. 
Assumptions  were  made  about  human  beings  and  solutions  to 
behavioral  problems  were  offered.   Improving  morale  was  a 
major  concern,  which  did  no  harm,  but  unfortunately  not  all 
desired  results  were  obtained. 
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Again  around  1970  the  behavioral  approach  had  a  parallel 
development  with  the  quantitative  approach.   A  more  broadly 
based  organizational  behavioral  approach  was  followed.   More 
direct  attention  is  devoted  to  organization  theory  and  organi- 
zational development.   This  approach  is  now  the  result  of 
interaction  between  the  human  being  and  the  formal  organization. 
3 .   The  Systems  Approach 

While  the  quantitative  and  behavioral  approaches  were 
going  their  separate  ways  the  systems  approach  appeared.  Since 
1960  this  approach  took  up  where  the  process  approach  left  off 
in  unifying  management  theory. 

The  theoretical  systems  approach  can  be  traced  back 
to  the  natural  and  physical  sciences  of  the  early  1950' s.  The 
application  to  management  has  been  much  more  recent.    The 
systems  approach  stresses  the  interrelatedness  and  inter- 
dependency  of  the  parts  to  the  whole. 

As  has  been  noted,  both  the  management  sciences  and 
organizational  behavioral  approaches  are  heading  back  toward 
the  main  path  of  systems.   In  management  science,  the  new 
emphasis  on  computer  applications  and  operations  management 
techniques  are  systems  based.   On  the  other  side  the  formal 
organization  is  viewed  as  a  system  consisting  of  structures, 
processes,  and  technology  and  is  composed  of  the  human  system 
which  contains  a  biological-physiological  structure,  psycho- 
logical processes,  and  a  personality. 

Only  time  will  tell  if  the  systems  approach  will 
unitify  the  quantitative  and  behavioral  approaches  to  management. 
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If  the  three  approaches  do  actually  merge  in  the  future  than 
something  different  from  the  sum  of  their  parts  may  emerge. 
The  future  theory  is  now  being  called  the  contingency  theory. 
4 .   The  Contingency  Theory 

The  contingency  theory,  sometimes  referred  to  as  situa- 
tional theory  is  starting  to  emerge  from  the  many  management 
theories.   This  theory  has  been  advanced  largely  by  practicing 
managers.   The  complex,  changing  environment  around  an  organiza- 
tion has  created  the  situation  of  "it  all  depends"  type  of 
management  philosophy. 

The  systems  approach  may  however  in  time  evolve  to 
meet  these  situational  theories.   An  open,  as  opposed  to  closed, 
systems  view  is  better  able  to  cope  with  increased  complexities 
and  environmental  influences   facing  managers  today.   Systems 
concepts  such  as  entropy  and  equif inability  are  quite  applicable 
to  the  present  managerial  situations. 

Nevertheless  a  mid-range  concept  that  falls  between 
"simplistic,  specific  principles"  and  "complex,  vague  notions" 
is  being  called  for  by  several  notable  authors  in  the  area. 
This  contingency  approach  "recognizes  the  complexity  involved 
in  managing  modern  organizations  but  uses  patterns  of  relation- 
ships and/or  configurations  of  subsystems  in  order  to  facilitate 
improved  practices"  [Kast,  1972]  .   This  author,  however,  feels 
that  flexibility  in  the  systems  approach  can  provide  the  de- 
sired results. 
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APPENDIX  C 
SYSTEM  MODELS 

Systems  may  be  represented  by  various  types  of  models.  The 
model,  as  used  in  this  thesis,  is  an  abstract  representation 
of  real  phenomena.   The  objective  of  the  model  is  not  to  por- 
tray or  identify  all  aspects  of  phenomena,  but  rather  to  point 
out  the  significant  elements  and  the  salient  interrelations. 
The  following  types  of  models  are  listed  in  order  of  their 
degree  of  abstraction. 

1.  Verbal  Models 

Verbal  models  rely  on  words  to  describe  the  elements 
and  the  interrelations  of  a  particular  system.   Difficulties 
are  encountered  however  in  the  ambiguities  and  semantic  problems 
often  encountered  in  verbal  communication. 

2 .  Schematic  Models 

Schematic  models  are  basically  a  picture  of  a  system 
and  are  typically  drawings  or  charts  that  present  systems 
elements  or  their  attributes  together  with  a  single  relation. 
Static  schematic  models  depict  a  set  of  elements  and  their 
relations  at  a  given  point  in  time;  such  as  maps,  bar  charts, 
Gantt  Charts,  the  breakeven  chart,  and  the  organizational 
chart.   Flow  schematic  models  are  schematic  models  that  protray 
flow  or  movement  of  some  description  among  the  system  elements, 
such  as  the  Program  Evaluation  and  Review  Technique  (PERT) , 
and  the  decision  tree.   Dynamic  system  models  are  schematic 
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models  that  depict  a  transformation  process.   Inputs  are  pro- 
cessed into  outputs  under  the  direction  of  a  controller.   It 
is  this  model  which  will  be  used  to  analyze  the  systems  approach 
to  test  and  evaluation  management. 

3 .  Iconic  Models 

Iconic  models  are  static  three-dimensional  representa- 
tions of  physical  objects  such  as  models  used  in  wind  tunnel 
testing. 

4 .  Analog  Models 

Analog  models  seek  to  describe  the  operational  charac- 
teristics of  other  systems  through  the  process  of  analogy. 
These  models  are  based  on  the  premise  that  the  behavior  of 
full-scale  systems  may  be  studied  by  examining  the  behavior 
of  other  systems  with  similiar  characteristics. 

5 .  Mathematical  Models 

Mathematical  models  are  symbolic  models  which  represent 
the  highest  level  of  abstraction  in  systems  model  construction. 
These  models  are  either  deterministic  or  stochastic  in  nature 
and  may  be  either  descriptive  or  optimizing  in  purpose. 
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